Experiment design identity and counts: excluded folders, duplicate columns, output folder (#82, #83, #98) - #130

Fixes #82, Fixes #83, Fixes #98.

Before-audit batch (triage docs/restart/triage-preprocessing-2026-11-02.md, bucket A). Binding: design v0.4 §3, §6, §9. Each claim below was reproduced by a test that fails on main (6ac81ad, after #126) and passes here.

Summary (by module)

#82: technical-repeat counts (dataset.py, report.py, notebook_render.py, notebook_design.py)

• biological_repeats() counted every technical-repeat folder, also one whose images were all excluded (dataset.py ~180). It now counts only folders with at least one image to analyse, so BiologicalRepeat.technical_repeat_count, the balance table and the design's technical_repeat_count are right (ctrl/rep1/techA + techB with techB's only image excluded: 1 technical repeat, not 2).
• BalanceTable gains excluded_images, excluded_folders and not_counted(). The CLI check balance text, the notebook's consistency card and its design card say how many images were excluded and name the folders not counted.
• The single part follows plan Q23 and is only documented: a technical-repeat folder named single is the level's value single and counts as no technical-repeat folder, like a missing level. There is a test for it, and the behaviour is unchanged.

#83: duplicate design columns (design.py, new design_values.py, run_inputs.py)

• Every design reader refuses a header that names a column twice or has a column with an empty or whitespace-only name. That covers read_experiment_design (and so load_design, notebook step 1, check, run --design and the design command) and read_confirmed_design. Names are compared ignoring case and outer whitespace. The DesignFormatError names the column and both positions, for example: the header names the column 'repeat_group' twice (columns 3 and 10), so it is not clear which value to read. Delete one of the two columns. Before this, the lenient reader kept the last column. The test that asserted that now asserts the refusal, with both conflicting and equal values. Nothing is rewritten on refusal: there are tests for load_design and for design --confirm-all.
• RunInputs.design() now reads with read_confirmed_design(path, expected_sha256=manifest.experiment_design_sha256) instead of the lenient reader. Any refusal becomes a RunInputsError, with the refusal as its __cause__. The SHA-256 is now taken from the bytes that are parsed, not from a separate hash followed by a second read.
• The header check and the value parsers (parse_bool, parse_count, is_date, DATE_FORMAT, DesignFormatError) move to design_values.py (pure). design re-exports them under the same names. The two readers share _check_rows (folder listed twice, control disagreeing). DesignFormatError.__module__ is now morphobricks_preprocessing.design_values.

#98: output folder change (notebook_folder.py, notebook_design.py, notebook_ui.py)

• _output_edited used to keep self.design, so the run wrote the old folder's confirmed design and controls into the new folder. Now, when the design card is open and the output folder resolves to another folder, the design is loaded again with _open_design(output_changed=True):
  • the design saved in the new folder is read, and an unreadable one is refused and kept as it is;
  • when the new folder has no design, the questions are asked again, unanswered and with no controls.
• Nothing is carried over, and nothing is written until the user answers. The field fires on every keystroke, so a path typed one character at a time creates no folder; a test checks this.
• The design card says what happened ("its experiment design was read" or "is asked again ... not carried over"), and "Not saved yet: your next answer saves it in ...". The run is gated as before. Changing back (A to B to A) reads A's design again.
• There is no "copy my design" action (owner, 2 November 2026: accepted). The user answers again or copies the file.

Tests

• New: tests/test_notebook_design_output.py. test_notebook_ui.py is already over the size limit, so the new widget tests go here; design_tree moves to tests/tiff_tree.py and both files share it.
• Extended: tests/test_dataset.py, tests/test_confirmed_design.py, tests/test_run_inputs.py.

Decisions (recorded in plan §12, renumbered 61–63 at the merge of #129)

• Q61 ([Sad Catapella B4][Scientific QC] Technical-repeat counts include excluded-only folders and omit explicit single folders #82): only technical-repeat folders with images to analyse are counted, and the text names what is excluded. single stays as Q23 says, documented only.
• Q62 ([Sad Catapella B4][Scientific identity] Lenient experiment-design reader silently selects last duplicate column #83, owner accepted 2 November 2026): duplicate or unnamed header columns are refused by every reader, compared ignoring case and outer whitespace (and invisible format characters, from the review). RunInputs.design() reads strictly.
• Q63 ([Sad Catapella B8][Scientific identity] Changing notebook output directory retains and can overwrite the wrong confirmed experiment design #98): the design belongs to its output folder. On a change it is read again or asked again, and never carried over.
• Contract §9.1, §11, §12 and §16, the user guide (§4 step 1, §8 messages), plan §10 and AGENTS.md are updated. The version stays 0.4.0rc3.

Review fixes (independent review)

• [Sad Catapella B4][Scientific identity] Lenient experiment-design reader silently selects last duplicate column #83, invisible characters (d4b7b14): the header check compared names ignoring case and outer whitespace only, so a second column written \ufeffcondition or Condition\u200b (a byte-order mark or zero-width space inside the header) looked new and the lenient reader (load_design, notebook step 1, check, run --design, design) silently read the first one. Invisible format characters (Unicode Cf) are now ignored when comparing, and a column of only such characters has no name. Plan Q62 and contract §12 say so; 3 new parametrized cases fail on the previous head.
• Output folder under a file (1b8bf87, pre-existing, owner-requested): somefile/out raised an uncaught NotADirectoryError from the run step's refresh, and a run click showed "unexpected problem". current_fingerprint treats a path under a file as gone; the output hint names the file in red; the design is not saved there; step 3 waits ("choose an output folder that is not a file"). Regression test test_an_output_folder_under_a_file_is_refused_with_a_message (fails on the previous head). User guide §4 step 1.
• 4ac8678: docstring line wrap (E501).
• main merged in twice, no rebase: PP-11b: the notebook's Make named-axis copies card #127 (b1c8157, no conflicts) and OME-Zarr axes and calibration: nothing guessed, nothing silent (#72, #73, #94) #129 (f7257b7: one conflict in plan §12, resolved by keeping OME-Zarr axes and calibration: nothing guessed, nothing silent (#72, #73, #94) #129's question 60 and renumbering this PR's 62–64 to 61–63, with every reference in the contract and plan §10).

Tests failing on main

16 regression tests fail with main's src and pass here:

• [Sad Catapella B4][Scientific QC] Technical-repeat counts include excluded-only folders and omit explicit single folders #82: 4 in test_dataset.py, plus the design-card test in test_notebook_design_output.py.
• [Sad Catapella B4][Scientific identity] Lenient experiment-design reader silently selects last duplicate column #83: 6 in test_confirmed_design.py (5 parametrized duplicate or unnamed header cases, plus the design command test), the updated 'condition' twice message case, and test_run_inputs.py::test_the_design_is_read_with_the_strict_reader.
• [Sad Catapella B8][Scientific identity] Changing notebook output directory retains and can overwrite the wrong confirmed experiment design #98: 3 widget tests in test_notebook_design_output.py: A to B to A with conflicting controls, a new empty folder typed character by character, and an unreadable design in the new folder.

The later locked-folder test (save failure) was not run against main.

Full gate (local, Python 3.11, .[dev,notebook,zarr], zarr 3.1.6)

• ruff check .: all checks passed. ruff format --check .: 179 files already formatted.
• pytest: 1797 passed, 1 skipped (implementer); review, after merging main (PP-11b: the notebook's Make named-axis copies card #127, OME-Zarr axes and calibration: nothing guessed, nothing silent (#72, #73, #94) #129): 1852 passed, 1 skipped (skip as root, expected). The skip is test_discovery.py:665, "running with permissions that ignore file modes", which is expected because the run is as root.
• python tools/check.py --rules: exit 0.

§9 figures (measured)

• Baseline entries: 44 → 43 (design.py _count removed).
• design.py: 964 → 926 lines. test_notebook_ui.py: 1537 → 1529.
• New files under the limits: design_values.py has 105 lines and is declared pure in pyproject. test_notebook_design_output.py has 162 lines.
• Other touched files: dataset.py 377, notebook_design.py 454, notebook_folder.py 406, run_inputs.py 592 lines.
• Ratcheted # noqa counts unchanged (ANN 54, PLR0913 3, C901 0, PLR0915 0). Total noqa in *.py is unchanged at 124: one E402 line was edited, none added.

Notes for review

• Plan question numbers are now 61–63 (after OME-Zarr axes and calibration: nothing guessed, nothing silent (#72, #73, #94) #129's 60); Links stay inside the experiment and masks folders; --log is checked before it opens (#81, #99) #128 renumbers after them.
• main was merged in (the PP-11c: conversion integrity (copy SHA-256, metadata epochs, exact CLI sizes) #126 conversion PR); the one conflict was in the run_inputs.py import lines. Not rebased.
• Widgets were tested headless only, not looked at in a real JupyterLab.
• The output folder under a file is fixed in this PR (see Review fixes).
• The merge commit carries git's default message, without the session trailers.

https://claude.ai/code/session_01G3qp9AxjpGPw9xfWAUJNqc

Generated by Claude Code